Task scheduling simulation system

ABSTRACT

The application provides a task scheduling simulation system, comprising a data preprocessing subsystem and a task scheduling subsystem. The data preprocessing subsystem filters the input cloud computing log information for abnormal data and extracts the running time of each task. The task scheduling subsystem enqueues or dequeues tasks from the batch task and real-time task running queues of each node, and keeps the tasks currently running in the cluster consistent with the actual production environment, and updates the number of CPU cores and the used and available memory capacity of each node according to resource requirement of each task. The mixed scheduling simulation of batch tasks and online tasks can be realized, and the resource simulation of the heterogeneous CPU core number and memory capacity of the cluster nodes can be simulated.

TECHNICAL FIELD

This application belongs to the field of cloud computing technology, andparticularly relates to a task scheduling simulation system.

DESCRIPTION OF RELATED ART

The task scheduling simulation system on the cloud computing platformcan use several machine nodes far less than the number of machines inthe production environment according to the log records of taskoperation in the production environment, to truly reflect the number oftasks to be processed by the resource management scheduling system at acertain time, as well as the changes such as the downtime and additionof cluster nodes, in order to provide researchers with an experimentalenvironment that is highly consistent with the real productionenvironment for the research of the scheduling algorithm, therebyproviding support for the effectiveness of the scheduling algorithm. Onthe other hand, compared with the case that the production environmentlacks the records of the worst task scheduling considerations, the taskscheduling simulation system can simulate the experimental scenariowhere the peak value of the number of submitted tasks or the resourceutilization of the entire cluster reaches a critical value through somespecial settings. Thus, it provides an operating environment for testingthe operating efficiency of the newly designed scheduling algorithm inthe worst case.

The Yarn simulation system (Scheduler Load Simulator, SLS) in the bigdata processing system Hadoop simulates and runs batch tasks based onMap-Reduce. The input of SLS running is the running log of batch tasks,including the running time of each task and the CPU and memory resourcesit requires. However, in the data center 24-hour log records publishedby some cloud computing platforms, it is the mixed scheduling andrunning of batch tasks and online tasks on some cloud computing datacenters. The SLS can only be aimed at the simulation system ofMap-Reduce single batch tasks in Hadoop.

SUMMARY 1. Technical Problems to be Solved

The Yarn simulation system (Scheduler Load Simulator, SLS) in the bigdata processing system Hadoop simulates and runs batch tasks based onMap-Reduce. The input of SLS running is the running log of batch tasks,including the running time of each task and the CPU and memory resourcesit requires. However, in the data center 24-hour log records publishedby some cloud computing platforms, it is the mixed scheduling andrunning of batch tasks and online tasks on some cloud computing datacenters. The SLS can only be aimed at the issues of the simulationsystem of Map-Reduce single batch tasks in Hadoop. This applicationprovides a task scheduling simulation system.

2. Technical Solution

In order to achieve the above objective, this application provides atask scheduling simulation system, which includes a data preprocessingsubsystem and a task scheduling subsystem.

The data preprocessing subsystem is used to filter the input cloudcomputing log information for abnormal data and extract the running timeof each task.

The task scheduling subsystem is used to enqueue or dequeue tasks fromthe batch task and real-time task running queues of each node, and keepthe number and status of tasks currently running in the clusterconsistent with the actual production environment, and update the numberof CPU cores and the used and available memory capacity of each nodeaccording to resource requirement of each task, to obtain the latesttopology of the overall cluster resource utilization.

Optionally, the data preprocessing subsystem includes a data exceptionand missing processing module, a task information counting module, aresource demand counting module, and a running time counting module.

The data exception and missing processing module is used to read thenative cloud computing cluster operation log to exclude abnormal data.

The task information counting module is used to count the taskinformation and the number of task instances of each submitted job.

The resource demand counting module is used to count the total requirednumber of CPU cores and the total required capacity of memory for eachjob.

The running time counting module is used to calculate the running timeof each task instance, and to counting the estimated running time ofeach job.

Optionally, the task information counting module, the resource demandcounting module and the running time counting module may simultaneouslystart 3 threads for parallel processing.

Optionally, the task scheduling subsystem includes a task operationinformation processing unit, a control unit, and a machine node eventprocessing unit.

The task operation information processing unit includes a task operationinformation loading module, a task event driving module and a taskscheduling algorithm module.

The control unit includes a simulator operation control module and amachine node resource information counting and collecting module.

The machine node event processing unit includes a machine node eventinformation module and a machine node event driving module.

Optionally, the task event driving module includes a batch task eventdriving submodule and an online task event driving submodule.

Optionally, the machine node event information module includes a nodeadding submodule and a node deleting submodule.

Optionally, the machine node event driving module includes a hash table.

3. Beneficial Effects

Compared with the prior art, the beneficial effects of the taskscheduling simulation system provided by this application are asfollows.

The task scheduling simulation system provided by this applicationrealizes the mixed scheduling simulation of batch tasks and online tasksby setting the data preprocessing subsystem and task schedulingsubsystem, and can also simulates the resource of the CPU core numberand memory capacity of cluster nodes heterogeneity.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a work flowchart of a data preprocessing subsystem of the taskscheduling simulation system according to the present application; and

FIG. 2 is a work flowchart of a task scheduling subsystem of the taskscheduling simulation system according to the present application.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, specific embodiments of the application will be describedin detail with reference to the accompanying drawings. According tothese detailed descriptions, those skilled in the art can clearlyunderstand and implement the application. Without departing from theprinciple of the present application, the features of differentembodiments can be combined to obtain new implementations, or somefeatures in some embodiments can be replaced to obtain other preferredimplementations.

A cluster is a group of independent computers interconnected by ahigh-speed network. They form a group and are managed as a singlesystem. When a client interacts with the cluster, the cluster operateslike an independent server. The cluster configuration is used to improveavailability and scalability. Compared with traditional high-performancecomputer technology, cluster technology can use various grades ofservers as nodes, thus the system cost is low, and it can achievehigh-speed computation, complete large-scale calculations, and has highresponsiveness, which can meet today's increasing demand for informationservices. The cluster technology is a general technology, its purpose isto solve the inadequacy of single-machine computing power, theinadequacy of IO capabilities, improve service reliability, obtain scalescalability, and reduce the operation and maintenance costs (operation,upgrade, maintenance costs) of the overall solution. In the situationsin which other technologies cannot achieve the above goals, or the abovegoals can be achieved, but the cost is too high, cluster technology canbe considered.

In the data center of the modern cloud computing platform, there aretens of thousands of cluster machines. For example, Google announced in2011 that the scale of cluster nodes could reach 12,500. In 2016, thenumber of machines in Microsoft's data center exceeded 50,000. At thesame time, in these large data centers, tens of thousands of jobs andtasks are scheduled and run every day. An effective job schedulingalgorithm can reasonably allocate jobs to the machine nodes that meetits running resource requirements, and significantly improve clusterresource utilization and task throughput per unit time. However,compared with the large-scale cluster nodes in the enterprise productionenvironment, the machine nodes of the relevant research team aresmaller, generally less than a few hundred, which is not enough to trulyand accurately restore the actual task scheduling status of theenterprise production environment. Therefore, in order to verify theeffectiveness of the new scheduling algorithm in the actual productionenvironment, it is particularly important to find a system that canfully simulate the real task scheduling in the production environmentand can run on a few machines.

The input of SLS does not include the hardware resource information ofthe cluster nodes and the dynamic addition and deletion logs of thecluster nodes during the operation period. The outputs of SLS operationare the memory usage at the JVM level and the CPU core usage on theentire cluster, which lacks the records of the resource utilizationstatistics information on a single machine node. SLS just treats thenode resources in the cluster as homogeneous machines with the samenumber of CPU cores and memory size when simulating them.

API is a calling interface left by the operating system to theapplication program. The application program makes the operating systemexecute the commands of the application program by calling the operatingsystem's API.

Hash table is a data structure that is directly accessed according tothe key value. In other words, it accesses the record by mapping the keyvalue to a location in the table to speed up the search. This mappingfunction is called a hash function, and the array which stores recordsis called a hash table.

Comma-Separated Values (CSV, sometimes referred to ascharacter-separated values, because the separator may not be a comma)have its files to store tabular data (numbers and text) in plain text.Plain text means that the file is a sequence of characters without datathat must be interpreted like binary numbers. A CSV file consists of anynumber of records, separated by a certain type of newline character.Each record is composed of fields, separated by other characters orstrings, the most common one is comma or tab.

This application provides a task scheduling simulation system, thesystem includes a data preprocessing subsystem and a task schedulingsubsystem.

The data preprocessing subsystem is used to filter the input cloudcomputing log information for abnormal data and extract the running timeof each task;

The task scheduling subsystem is used to enqueue or dequeue tasks fromthe batch task and real-time task running queues of each node, and keepthe number and status of tasks currently running in the clusterconsistent with the actual production environment, and update the numberof CPU cores and the used and available memory capacity of each nodeaccording to resource requirement of each task, to obtain the latesttopology of the overall cluster resource utilization.

Optionally, the data preprocessing subsystem includes a data exceptionand missing processing module, a task information counting module, aresource demand counting module, and a running time counting module.

The data exception and missing processing module is used to read thenative cloud computing cluster operation log to exclude abnormal data.

The task information counting module is used to count the taskinformation and the number of task instances of each submitted job.

The resource demand counting module is used to count the total requirednumber of CPU cores and the total required capacity of memory for eachjob.

The running time counting module is used to calculate the running timeof each task instance, and to counting the estimated running time ofeach job.

The input of the data preprocessing subsystem is the native cloudcomputing task running log, and the output is the native cloud computingtask log information and the above statistical information. The abovestatistical information can be obtained by the user through the APIprovided by the system. The return format of the information is j son.The hardware requirement resources of the task can be displayed on theweb page according to the information and the third-party chartvisualization tool.

Optionally, the task information counting module and the resource demandcounting module, and the running time counting module can simultaneouslystart 3 threads for parallel processing.

The task scheduling subsystem includes a task operation informationprocessing unit, a control unit, and a machine node event processingunit.

The task operation information processing unit includes a task operationinformation loading module, a task event driving module and a taskscheduling algorithm module.

The control unit includes a simulator operation control module, and amachine node resource information counting and collecting module.

The machine node event processing unit includes a machine node eventinformation module and a machine node event driving module.

The task event driving module includes a batch task event drivingsubmodule and an online task event driving submodule.

The task operation information loading module is used to:

S101: classifying the batch tasks and online tasks.

S102: adding the time stamp in the task record as the key value and thetask record as the value to the Leveldb database with higher sequentialread and write performance supported by the simulator.

S103: loading the data information of the machine node to the mapdisordered set in the memory of the simulator, wherein the key is thetimestamp of the machine node event, and the value is the data record ofthe machine node.

The machine node event driving module is used to perform:

S201: in response to an increase or failure event of related machinenodes, the simulator updates the globally available machine nodes of thecluster using an event driving model, according to the wal-clock time ofthe current simulator.

S202: outputting the update information of the machine node to therelevant directory using the Google log recording module.

The batch task event driving module is used to perform:

S301: executing event processing in response to the running event state(prepare, wait, terminate, fail, cancel, interrupt) of the batch taskinstance using the event driving model, according to the wal-clock timeof the current simulator. If the batch task instance is in the waitstate, the S5 task algorithm scheduling module is triggered to executethe relevant algorithm for task scheduling. If the task instance is inthe fail state, terminate state or cancel state, the resourceinformation on the running node is updated.

The online task event driving module is used to perform:

S401: using an event-driven model, based on the wal-clock time of thecurrent simulator: if the event state of the online task is in thegeneration state, the S5 task algorithm scheduling module is triggeredto perform task scheduling; if the event state of the online task is inthe removal state, the related machine node resource usage is updated.

The task scheduling algorithm module is used to perform:

S501: integrating different task scheduling algorithms into thescheduling algorithm library of the simulator using the plug-in insoftware design mode. The user can specify the scheduling algorithm usedin this operation of the simulator through the configuration file xml.

The machine node resource information counting and collecting module isused to perform:

S601: dynamically calculating the number of CPU cores and memorycapacity usage of each node at a certain time according to the number oftasks running on each node and the resource consumption of the tasks.

S602: if the user needs to analyze the resource utilization rate of thecluster at each moment in real time, the machine node resourceinformation counting and collecting module can output the resourceutilization status on the node to CSV file at a certain interval (suchas 5 seconds) after receiving the user's instruction.

The simulator operation control module is used to perform:

S701: setting the start time point and end time point of the wal-clockthat the simulator runs. These two time points correspond to two timepoints in the Alibaba Cloud log.

S702: setting the acceleration ratio of the simulator operation.

Optionally, the machine node event information module includes a nodeadding submodule and a node deleting submodule.

Optionally, the machine node event driving module includes a hash table.

The task scheduling subsystem first sets the task scheduling time periodrequired by the simulator to simulate the cloud computing data centerthrough the simulator operation control module, starts the simulatoroperation. Then the task operation information loading module loads taskinformation that needs to be simulated from the output data of the datapreprocessing subsystem, loads new machine node information in real timethrough the machine node event driving module, manages the runningstatus of tasks through the batch task event driving module and onlinetask event driving module, loads the specified scheduling algorithmthrough the task scheduling algorithm module and schedules the tasks inthe wait state, and calculates the number of CPU cores and memory usageof each node in real time through the machine node resource informationcounting and collecting module, and outputs them to the specified outputdirectory.

Example

This application relates to a task scheduling simulation system for acluster environment, and it is explained in detail with Alibaba Cloud asthe object:

Refer to FIGS. 1-2, the data preprocessing subsystem is shown first. Asshown in FIG. 1, the input of the data preprocessing subsystem is the24-hour running log published by Alibaba Cloud, and the output is thepreprocessed CSV file as the input data of the subsequent simulatorsystem. The preprocessing is divided into 4 modules. The data exceptionhandling module reads 4 running logs of the native Alibaba Cloud clusterfor exception handling. Exception handling operations mainly includeremoving task instance records and online task records whose end time isless than the start time. For the lack of information on the resourcerequirements of the batch task instances, the average value of theresource requirements of the unified task instances is used to fill. Forexample, if the CPU core numbers application record of a certain batchtask instance is missing, an average value of the CPU core numbersapplication of all other task instances with the same task account iscalculated, and the missing value is replaced with this average value.

The task information counting module is to count the task information ofeach submitted job, including counting the number of tasks owned by eachtask, and forming a map wherein the job ID and task ID set maps. The keyis the job ID and the value is the ID set of the task.

The resource demand counting module first sums the required number ofCPU cores and memory capacity of each task instance according to asingle task, and then counts the total required number of CPU cores anddemanded memory capacity of this task. Then, the required number of CPUcores and demanded memory capacity of each task instance according to asingle task are summed, and the total required number of CPU cores andtotal demanded memory capacity of this job are calculated.

The running time counting module calculates the running time of eachbatch task or online task instance from the log records. Since the starttime of the task instance may be earlier than the start time of AlibabaCloud log sampling, and the end time of the task instance may be laterthan 24 hours, there are two situations. First, if the task instancestarts to run earlier than midnight, modify the start time of the taskinstance to 0 seconds. Second, if the end time of the task instance islater than 24 hours, modify the end time of the task instance to an intinteger Maximum value. Finally, the running time of each task instanceis calculated as the end time of the task instance minus the start time,in seconds.

Finally, the new log records generated by the task information countingmodule, resource demand counting module, and running time countingmodules are output to the intermediate data CSV file. The above threemodules can starts 3 threads for parallel processing at the same time.

2. Task Scheduling Subsystem

The specific workflow of the task scheduling subsystem is shown in FIG.2. Firstly, the user inputs the time period during which the simulatorneeds to simulate the task running in the cloud platform, such as 0:00to 12:59. This time period information is used for the initialization ofthe simulator operation control module. After initialization, thesimulation clock starts to run. Secondly, the simulator operationcontrol module starts the machine node event driving module and the taskevent driving module, and then reads the intermediate data CSV fileoutput by the data preprocessing subsystem line by line according to thecurrent simulation clock. If the read information belongs to the machinenode event file, the information is sent to the machine node eventdriving module which is responsible for parsing. The machine node eventinformation module may be divided into two types: node adding submoduleand node deleting submodule. The machine node event driving module usesa hash table to record the machine node information in the currentcluster. Therefore, when node adding and deleting event informationneeds to be processed, the machine node event driving module operatesthe hash table to add or delete cluster nodes, in order to simulate thecurrent number of cluster machine nodes and resource conditionsconsistent with actual production environment logs. On the other hand,the task operation information loading module is responsible for loadingdata from the intermediate data CSV file into the memory map datastructure. The simulator operation control module sends the task eventinformation of the current clock from the map to the task event drivingmodule for processing. The task event driving module analyzes these taskevent information, obtains the CPU and memory requirements of each task,generates a directed acyclic graph of the task, and submits it to thetask scheduling algorithm module for resource allocation and taskscheduling.

Before running the task scheduling algorithm, the simulator operationcontrol module notifies the machine node resource information countingand collecting module to collect the resource utilization of each node,including the number of the remaining CPU cores and allocatable memorycapacity, and finally to update the resource utilization topology of theentire cluster. The task algorithm scheduling module takes this resourceutilization topology as input data, loads the user-specified schedulingalgorithm code from the algorithm scheduling library, and runs the taskscheduling program. At the same time, the simulator operation controlmodule records the start time and end time of the operation of thescheduling program, calculates the running time of the schedulingprogram, and returns it to the user as the running efficiency of thescheduling algorithm. When the task scheduling program is executed, thematching information of the task and the node is obtained. Based on thisinformation, the task event driving module updates the node task queuingtable it maintains, that is, enqueue or dequeue tasks from the batchtask and real-time task running queues of each node, so as to maintainthe current number and status of running tasks in the cluster consistentwith that of the actual production environment. On the other hand, themachine node resource information counting and collecting module rescansthe task running queue in each node, and updates the number of CPU coresand the used and available amounts of the memory capacity of each nodeaccording to the resource requirements of each task. Finally, theresource utilization topology of the entire cluster is updated.

In the data center of Alibaba Cloud, machine nodes show heterogeneity inthe number of CPU cores and memory capacity due to the replacement. Forthis reason, the log information published by the Alibaba Cloud DataCenter not only records the number of CPU cores, memory capacity anddisk capacity of the machine node, but also records the time stamp ofthe joining or downtime of each machine node. Therefore, the taskscheduling of the cluster at a certain moment will be constrained by thenumber of CPU cores and memory capacity available on each machine node.Since SLS does not consider the hardware resources of actual machinenodes, it treats the CPU and memory resources of all machine nodes assame type, therefore, SLS cannot accurately and fully simulate thescheduling of Alibaba Cloud tasks. Based on the 24-hour cloud computingplatform cluster task operation log published by Alibaba Cloud, thissystem realizes the process of simulating the task submission,scheduling, running and termination of Alibaba Cloud cluster nodes on asingle machine node. Moreover, at a certain time within 24 hours, thesystem can simulate the utilization of CPU and memory resources on eachmachine node according to the number of tasks running on each machinenode and its life cycle status.

Although the application has been described above with reference tospecific embodiments, those skilled in the art should understand thatmany modifications can be made to the configuration and detailsdisclosed in the application without departing the principles and scopedisclosed in the application. The protection scope of this applicationis determined by the appended claims, and the claims are intended tocover all the modifications included in the literal meaning or scope ofequivalents of the technical features in the claims.

1. A task scheduling simulation system, the system includes a datapreprocessing subsystem and a task scheduling subsystem; the datapreprocessing subsystem is used to filter the input cloud computing loginformation for abnormal data and extract running time of each task; thetask scheduling subsystem is used to enqueue or dequeue tasks from thebatch task and real-time task running queues of each node, and keep thenumber and status of tasks currently running in the cluster consistentwith the actual production environment, and update the number of CPUcores, and the used and available memory capacity of each node accordingto resource requirement of each task, to obtain the latest topology ofthe overall cluster resource utilization.
 2. The task schedulingsimulation system of claim 1, wherein the data preprocessing subsystemcomprises a data exception and missing processing module, a taskinformation counting module, a resource demand counting module, and arunning time counting module; the data exception and missing processingmodule is used to read the native cloud computing cluster operation logto exclude abnormal data; the task information counting module is usedto count the task information and the number of task instances of eachsubmitted job; the resource demand counting module is used to count thetotal required number of CPU cores and the total required capacity ofmemory for each job; the running time counting module is used tocalculate the running time of each task instance, and to counting theestimated running time of each job.
 3. The task scheduling simulationsystem of claim 2, wherein in that the task information counting module,the resource demand counting module, and the running time countingmodule simultaneously start 3 threads for parallel processing.
 4. Thetask scheduling simulation system of claim 1, wherein the taskscheduling subsystem comprises a task operation information processingunit, a control unit, and a machine node event processing unit; the taskoperation information processing unit comprises a task operationinformation loading module, a task event driving module and a taskscheduling algorithm module; the control unit comprises a simulatoroperation control module, and a machine node resource informationcounting and collecting module; the machine node event processing unitcomprises a machine node event information module and a machine nodeevent driving module.
 5. The task scheduling simulation system of claim4, wherein the task event driving module comprises a batch task eventdriving submodule and an online task event driving submodule.
 6. Thetask scheduling simulation system of claim 4, wherein the machine nodeevent information module comprises a node adding submodule and a nodedeleting submodule.
 7. The task scheduling simulation system of claim 4,wherein the machine node event driving module comprises a hash table.